Method for computational microscopic layer separation

ABSTRACT

A microscope for computational microscopic layer separation may include an imaging device that includes a lens and an image sensor, an illumination system for illuminating a sample, and an actuator to adjust an axial position of a focal plane with respect to the sample. The microscope may also include a processor operatively coupled to the imaging device and the illumination system. The processor may be configured to measure, using the image sensor and the illumination system, optical aberrations of the imaging device at the axial position, and determine whether to adjust the focal plane with respect to the sample in response to the one or more optical aberrations. Various other systems and methods are also disclosed.

RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Patent Application No. 62/935,791, filed Nov. 15, 2019, andtitled “COMPUTATIONAL MICROSCOPIC LAYER SEPARATION,” which isincorporated, in its entirety, by this reference.

BACKGROUND

Microscopy is an important field with many applications, such as healthcare and metrology. Work in relation to the present disclosure suggeststhat the prior approaches to focusing microscopes on samples andscanning microscope slides can be less than ideal in at least somerespects. For example, with some of the prior approaches the amount oftime to scan a sample can be longer than would be ideal. Also, automatedapproaches for setting the focus of the microscope can set the focusincorrectly in at least some instances.

In whole-slide digital microscopy, an area in the slide may be scannedby acquiring a series of fields of view (“FOVs”). When preparing aslide, a high refractive index material can be applied on top of thespecimen in order to protect it from outside dirt, as well as securingit in place and flattening the sample. These materials may include aglass coverslip, film plastic coverslip, oils, glues or liquidcoverslip. The use of a coverslip may result in two layers of data:“legitimate” data in between the slide and the coverslip, and“undesired” data above the coverslip. The latter may include dirt suchas dust, glue residues, fingerprints, etc.

With prior approaches, dirt on top of the cover slip may be mistaken forlegitimate data and may cause auto-focus algorithms to lock onto it inat least some instances. This problem can be more prevalent when thesample scanned is sparse, since in at least some FOVs dirt may be thedominant component in the image. This focusing on an inappropriatelayer, such as dirt on top of a slide, can result in degraded microscopedata or increased scan times in at least some instances.

Another prior approach for distinguishing between the two layers is toperform “z-stack” scanning through the focal planes of a FOV and findingthe focal planes of the data along the way, e.g. by using sharpnessmetrics to determine where features are the sharpest throughout thescan. However, this approach can result in scan times that are longerthan would be ideal and datasets that are larger than would be ideal inat least some instances.

Another prior approach for identification of the relevant sample layeris by using a special coverslip or slide, where several markings areprinted at the interface between the coverslip and the slide. Theprinted markings could be recognized automatically and help the systemreach the correct focal plane. Although helpful for identifying thecorrect focal plane, this approach relies on additional hardware, aswell as algorithms that search for and lock onto the printed markingswhich may slow down the scan.

In light of the forgoing, there is a need for improved systems andmethods of focusing microscopes that ameliorate at least some of theaforementioned limitations. Ideally these systems and methods wouldfocus the microscope at an appropriate sample plane, without relying onadditional hardware or appreciably increasing the time to scan a sample.

SUMMARY

In some embodiments, the systems and methods described herein provideimproved scanning of microscope samples with improved placement of focalplanes for data acquisition within samples, which can decrease scantimes and the size of data obtained with a scan of a microscope slide.In some embodiments, one or more optical aberrations is measured and thefocal plane adjusted in response to the aberrations. While theaberrations can be measured in many ways, in some embodiments the one ormore aberrations is measured by illuminating the sample at a pluralityof angles and an amount of the one or more aberrations determined inresponse to a shift among the plurality of images. The microscope maycomprise an imaging device that images the sample through a coverslipwith decreased aberrations as compared to material on the coverslip suchas dirt. In some embodiments, the processor is configured withinstructions to advance a focal plane toward a sample or not to advancethe focal plane in response to an amount of the one or more aberrations.

In some embodiments, a microscope for computational microscopic layerseparation comprises an imaging device that includes a lens and an imagesensor, an illumination system for illuminating a sample, and anactuator to adjust an axial position of a focal plane with respect tothe sample. The microscope may also include a processor operativelycoupled to the imaging device and the illumination system. The processormay be configured to measure, using the image sensor and theillumination system, optical aberrations of the imaging device at theaxial position, and determine whether to adjust the focal plane withrespect to the sample in response to the one or more opticalaberrations.

INCORPORATION BY REFERENCE

All patents, applications, and publications referred to and identifiedherein are hereby incorporated by reference in their entirety and shallbe considered fully incorporated by reference even though referred toelsewhere in the application.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the features, advantages and principles of thepresent disclosure will be obtained by reference to the followingdetailed description that sets forth illustrative embodiments, and theaccompanying drawings of which:

FIG. 1 shows a diagram of an exemplary microscope, in accordance withsome embodiments;

FIGS. 2A-B show diagrams of a focal plane of an exemplary microscope, inaccordance with some embodiments;

FIG. 3 shows a flow chart of a method of computational microscopic layerseparation, in accordance with some embodiments; and

FIG. 4 shows a flow chart of a process of computational microscopiclayer separation, in accordance with some embodiments.

DETAILED DESCRIPTION

The following detailed description and provides a better understandingof the features and advantages of the inventions described in thepresent disclosure in accordance with the embodiments disclosed herein.Although the detailed description includes many specific embodiments,these are provided by way of example only and should not be construed aslimiting the scope of the inventions disclosed herein.

The present disclosure is generally directed to systems and methods forcomputational microscopic layer separation that may allow for correctingfocus errors related to aberrations and to identify the sample forscanning, as compared to other material such as dirt on a coverslip. Aswill be explained in greater detail below, embodiments of the instantdisclosure may be configured to measure one or more optical aberrationsof an imaging device at an axial position. The focal plane of theimaging device may be adjusted with respect to the sample in response tothe optical aberrations. The systems and methods described herein mayimprove the field of digital microscopy by being able to adjust thefocal plane without the use of specialized hardware or microscopedesign.

The presently disclosed systems and methods are well suited for use withprior microscopes such as computational microscopes, and can beincorporated into and combined with such prior systems. An exemplarymicroscope suitable for modification in accordance with the presentdisclosure is described in U.S. Pat. No. 10,705,326, granted on Jul. 7,2020, entitled “Autofocus system for a computational microscope”, theentire disclosure of which is incorporated herein by reference.

Tomography refers generally to methods where a three-dimensional (3D)sample is sliced computationally into several 2D slices. Confocalmicroscopy refers to methods for blocking out-of-focus light in theimage formation which improves resolution and contrast but tends to leadto focusing on a very thin focal plane and small field of view. Althoughreference is made to computational microscopy, the presently disclosedsystems and methods are well suited for use with many types ofmicroscopy such as one or more of a high definition microscope, adigital microscope, a computational microscope, a 3D microscope, a phaseimaging microscope, a phase contrast microscope, a dark fieldmicroscope, a differential interference contrast microscope, alightsheet microscope, a confocal microscope, a holographic microscope,or a fluorescence-based microscope

The following will provide, with reference to FIGS. 1-4, detaileddescriptions of computational microscopic layer separation. FIGS. 1 and2A-B illustrate a microscope and various microscope configurations.FIGS. 3 and 4 illustrate an exemplary processes for computationalmicroscopic layer separation.

FIG. 1 is a diagrammatic representation of a microscope 100 consistentwith the exemplary disclosed embodiments. In some embodiments, themicroscope comprises a device or instrument for magnifying an objectwhich is smaller than easily observable by the naked eye, i.e., creatingan image of an object for a user where the image is larger than theobject. In some embodiments, the microscope comprises an opticalmicroscope that uses light in combination with an optical system formagnifying an object. An optical microscope may be a simple microscopehaving one or more magnifying lens. In some embodiments, the microscopecomprises a computational microscope, which comprises an image sensorand image-processing algorithms to enhance or magnify the object's sizeor other properties, such as resolving power of the microscope. Thecomputational microscope may be a dedicated device or created byincorporating software and/or hardware with an existing opticalmicroscope to produce high-resolution digital images. As shown in FIG.1, microscope 100 comprises an image capture device 102, a focusactuator 104, a controller 106 connected to memory 108, an illuminationassembly 110, and a user interface 112. An example usage of microscope100 may be capturing images of a sample 114 mounted on a stage 116located within the field-of-view (FOV) of image capture device 102,processing the captured images, and presenting on user interface 112 amagnified image of sample 114.

Image capture device 102 may be used to capture images of sample 114. Insome embodiments, the image capture device comprises a device thatrecords the optical signals entering a lens as an image or a sequence ofimages. The optical signals may be in the near-infrared, infrared,visible, and ultraviolet spectrums. Examples of an image capture devicecomprise a CCD camera, a CMOS camera, a color camera, a photo sensorarray, a video camera, a mobile phone equipped with a camera, a webcam,a preview camera, a microscope objective and detector, etc. Someembodiments may comprise only a single image capture device 102, whileother embodiments may comprise two, three, or even four or more imagecapture devices 102. In some embodiments, image capture device 102 maybe configured to capture images in a defined field-of-view (FOV). Also,when microscope 100 comprises several image capture devices 102, imagecapture devices 102 may have overlap areas in their respective FOVs.Image capture device 102 may have one or more image sensors (not shownin FIG. 1) for capturing image data of sample 114. In other embodiments,image capture device 102 may be configured to capture images at an imageresolution higher than VGA, higher than 1 Megapixel, higher than 2Megapixels, higher than 5 Megapixels, 10 Megapixels, higher than 12Megapixels, higher than 15 Megapixels, or higher than 20 Megapixels. Inaddition, image capture device 102 may also be configured to have apixel size smaller than 15 micrometers, smaller than 10 micrometers,smaller than 5 micrometers, smaller than 3 micrometers, or smaller than1.6 micrometer.

In some embodiments, microscope 100 comprises focus actuator 104. Insome embodiments, the focus actuator comprises a device capable ofconverting input signals into physical motion for adjusting the relativedistance between sample 114 and image capture device 102. Various focusactuators may be used, including, for example, linear motors,electrostrictive actuators, electrostatic motors, capacitive motors,voice coil actuators, magnetostrictive actuators, etc. In someembodiments, focus actuator 104 may comprise an analog position feedbacksensor and/or a digital position feedback element. Focus actuator 104 isconfigured to receive instructions from controller 106 in order to makelight beams converge to form a clear and sharply defined image of sample114. In the example illustrated in FIG. 1, focus actuator 104 may beconfigured to adjust the distance by moving image capture device 102.

However, in other embodiments, focus actuator 104 may be configured toadjust the distance by moving stage 116, or by moving both image capturedevice 102 and stage 116. Microscope 100 may also comprise controller106 for controlling the operation of microscope 100 according to thedisclosed embodiments. Controller 106 may comprise various types ofdevices for performing logic operations on one or more inputs of imagedata and other data according to stored or accessible softwareinstructions providing desired functionality. For example, controller106 may comprise a central processing unit (CPU), support circuits,digital signal processors, integrated circuits, cache memory, or anyother types of devices for image processing and analysis such as graphicprocessing units (GPUs). The CPU may comprise any number ofmicrocontrollers or microprocessors configured to process the imageryfrom the image sensors. For example, the CPU may comprise any type ofsingle- or multi-core processor, mobile device microcontroller, etc.Various processors may be used, including, for example, processorsavailable from manufacturers such as Intel®, AMD®, etc. and may comprisevarious architectures (e.g., x86 processor, ARM®, etc.). The supportcircuits may be any number of circuits generally well known in the art,including cache, power supply, clock and input-output circuits.Controller 106 may be at a remote location, such as a computing devicecommunicatively coupled to microscope 100.

In some embodiments, controller 106 may be associated with memory 108used for storing software that, when executed by controller 106,controls the operation of microscope 100. In addition, memory 108 mayalso store electronic data associated with operation of microscope 100such as, for example, captured or generated images of sample 114. In oneinstance, memory 108 may be integrated into the controller 106. Inanother instance, memory 108 may be separated from the controller 106.

Specifically, memory 108 may refer to multiple structures orcomputer-readable storage mediums located at controller 106 or at aremote location, such as a cloud server. Memory 108 may comprise anynumber of random access memories, read only memories, flash memories,disk drives, optical storage, tape storage, removable storage and othertypes of storage.

Microscope 100 may comprise illumination assembly 110. In someembodiments, the illumination assembly comprises a device or systemcapable of directing light to illuminate sample 114, such asillumination at a plurality of angles.

Illumination assembly 110 may comprise any number of light sources, suchas light emitting diodes (LEDs), LED array, lasers, and lamps configuredto emit light, such as a halogen lamp, an incandescent lamp, or a sodiumlamp. For example, illumination assembly 110 may comprise a Kohlerillumination source. Illumination assembly 110 may be configured to emitpolychromatic light. For instance, the polychromatic light may comprisewhite light.

In one embodiment, illumination assembly 110 may comprise only a singlelight source. Alternatively, illumination assembly 110 may comprisefour, sixteen, or even more than a hundred light sources organized in anarray or a matrix. In some embodiments, illumination assembly 110 mayuse one or more light sources located at a surface parallel toilluminate sample 114. In other embodiments, illumination assembly 110may use one or more light sources located at a surface perpendicular orat an angle to sample 114.

In addition, illumination assembly 110 may be configured to illuminatesample 114 in a series of different illumination conditions. In oneexample, illumination assembly 110 may comprise a plurality of lightsources arranged in different illumination angles, such as atwo-dimensional arrangement of light sources. In this case, thedifferent illumination conditions may comprise different illuminationangles. For example, FIG. 1 depicts a beam 118 projected from a firstillumination angle α1, and a beam 120 projected from a secondillumination angle α2. In some embodiments, first illumination angle α1and second illumination angle α2 may have the same value but oppositesign. In other embodiments, first illumination angle α1 may be separatedfrom second illumination angle α2. However, both angles originate frompoints within the acceptance angle of the optics. In another example,illumination assembly 110 may comprise a plurality of light sourcesconfigured to emit light in different wavelengths. In this case, thedifferent illumination conditions may comprise different wavelengths.For instance, each light source may be configured to emit light with afull width half maximum bandwidth of no more than 50 nm so as to emitsubstantially monochromatic light. In yet another example, illuminationassembly 110 may be configured to use a number of light sources atpredetermined times. In this case, the different illumination conditionsmay comprise different illumination patterns. For example, the lightsources may be arranged to sequentially illuminate the sample atdifferent angles to provide one or more of digital refocusing,aberration correction, or resolution enhancement. Accordingly andconsistent with the present disclosure, the different illuminationconditions may be selected from a group including: different durations,different intensities, different positions, different illuminationangles, different illumination patterns, different wavelengths, or anycombination thereof.

Consistent with disclosed embodiments, microscope 100 may comprise, beconnected with, or in communication with (e.g., over a network orwirelessly, e.g., via Bluetooth) user interface 112. In someembodiments, a user interface comprises a device suitable for presentinga magnified image of sample 114 or any device suitable for receivinginputs from one or more users of microscope 100. FIG. 1 illustrates twoexamples of user interface 112. The first example is a smartphone or atablet wirelessly communicating with controller 106 over a Bluetooth,cellular connection or a Wi-Fi connection, directly or through a remoteserver. The second example is a PC display physically connected tocontroller 106. In some embodiments, user interface 112 may compriseuser output devices, including, for example, a display, tactile device,speaker, etc. In other embodiments, user interface 112 may comprise userinput devices, including, for example, a touchscreen, microphone,keyboard, pointer devices, cameras, knobs, buttons, etc. With such inputdevices, a user may be able to provide information inputs or commands tomicroscope 100 by typing instructions or information, providing voicecommands, selecting menu options on a screen using buttons, pointers, oreye-tracking capabilities, or through any other suitable techniques forcommunicating information to microscope 100. User interface 112 may beconnected (physically or wirelessly) with one or more processingdevices, such as controller 106, to provide and receive information toor from a user and process that information. In some embodiments, suchprocessing devices may execute instructions for responding to keyboardentries or menu selections, recognizing and interpreting touches and/orgestures made on a touchscreen, recognizing and tracking eye movements,receiving and interpreting voice commands, etc.

Microscope 100 may also comprise or be connected to stage 116. Stage 116comprises any horizontal rigid surface where sample 114 may be mountedfor examination. Stage 116 may comprise a mechanical connector forretaining a slide containing sample 114 in a fixed position. Themechanical connector may use one or more of the following: a mount, anattaching member, a holding arm, a clamp, a clip, an adjustable frame, alocking mechanism, a spring or any combination thereof. In someembodiments, stage 116 may comprise a translucent portion or an openingfor allowing light to illuminate sample 114. For example, lighttransmitted from illumination assembly 110 may pass through sample 114and towards image capture device 102. In some embodiments, stage 116and/or sample 114 may be moved using motors or manual controls in the XYplane to enable imaging of multiple areas of the sample.

FIG. 2A illustrates a basic schematic of an exemplary microscopeaccording to some embodiments. FIG. 2A illustrates a microscope 200(which may correspond to microscope 100), that may include a imagecapture device 202 (which may correspond to image capture device 102), afocus actuator 204 (which may correspond to focus actuator 104), acontroller 206 (which may correspond to controller 106) connected to amemory 208 (which may correspond to memory 108), an illuminationassembly 210 (which may correspond to illumination assembly 110), a tubelens 232, an objective lens 234, a sample 214 mounted on a stage 216(which may correspond to stage 116), and a lateral actuator 236. Sample214 may include a specimen 246 (e.g., the actual sample to be scanned),between a slide 242 and a coverslip 244 (having a thickness 245), andpotentially dirt 248 on coverslip 244. Tube lens 232 and objective lens234 may function in unison to focus light of a focal plane 250 (whichmay be determined based on a position of objective lens 234 as adjustedby focus actuator 204) of sample 214 in an FOV of image capture device202. Tube lens 232 may comprise a multi-element lens apparatus in a tubeshape, which focuses light in conjunction with objective lens 234.Lateral actuator 236 may comprise a motor or other actuator describedherein that may be capable of physically moving stage 226 laterally inorder to adjust a relative lateral position between sample 214 and imagecapture device 202. In some examples, focus actuator 204 and/or lateralactuator 236 may comprise a coarse actuator for long range motion and afine actuator for short range motion. The coarse actuator may remainfixed while the fine focus actuator of focus actuator 204 adjusts thefocal distance and lateral actuator 236 moves the lateral position ofsample 214 for the movement paths. The coarse actuator may comprise astepper motor and/or a servo motor, for example. The fine actuator maycomprise a piezo electric actuator. The fine actuator may be configuredto move sample 214 by a maximum amount within a range from 5 microns to500 microns. The coarse actuator may be configured to move sample 214 bya maximum amount within a range from 1 mm to 100 mm.

Stage 216 may be configured to hold sample 214. Illumination assembly210 may comprise an illumination source configured to illuminate sample214. Image capture device 202 may be configured to capture multipleimages or frames of sample 214 within an FOV of image capture device 202at focal plane 250. Lateral actuator 236 may be configured to change arelative lateral position between image capture device 202 and an imagedportion of sample 214 within the FOV of image capture device 202 foreach of the multiple images. Focus actuator 204 may be configured toadjust a focal distance (e.g., focal plane 250) between sample 214 andimage capture device 202 between each of the multiple captured images.Controller 206, may comprise a processor operatively coupled to lateralactuator 236, focus actuator 204, image capture device 202, and/orillumination assembly 210 in order to move sample 214 laterally relativeto the FOV and capture an area of sample 214 one or more times. In someexamples, controller 206 may be configured to apply each of multiplelight colors (using illumination assembly 210) for one or more captures.

Although the examples herein describe adjusting the relative lateralposition by physically moving stage 216, in other embodiments therelative lateral position may be adjusted in other ways, includingmoving/shifting one or more of image capture device 202, tube lens 232,objective lens 234, sample 214, and/or stage 216. Likewise, although theexamples herein describe adjusting the focal distance by physicallymoving objective lens 234, in other embodiments the focal distance maybe adjusted in other ways, including moving/shifting one or more ofimage capture device 202, tube lens 232, objective lens 234, sample 214,and/or stage 216.

As seen in FIG. 2A, focal plane 250 may lie within dirt 248, which maynot be desirable. Dirt 248 may lie on coverslip 244. Specimen 246, whichmay be a sample prepared for imaging, may be protected between coverslip244 and slide 242. Coverslip 244 may comprise a solid or liquidcoverslip comprising a transparent material such that image capturedevice 202 may capture specimen 246 through coverslip 244. However,because coverslip 244 may be transparent, focal plane 250 lying withindirt 248 on top of coverslip 244 may appear as legitimate data (e.g.,may be confused for specimen 246). In some embodiments, the imagingdevice is configured to measure the sample with decreased aberrationsthrough the coverslip as opposed to material located on top of thecoverslip such as dirt, which can be related to the index of refractionof air as compared to the index of refraction of the coverslip.

FIG. 2B shows microscope 200, which may be correctly focused on sample214 such that focal plane 250 lies within specimen 246. As will bedescribed further below, microscope 200 may detect that focal plane 250lies within dirt 248 (in FIG. 2A), and correct, via controlling focusactuator 204 to move objective lens 234, focal plane 250 by shiftingfocal plane 250 down into specimen 246, as seen in FIG. 2B. Although thepresent disclosure describes shifting focal plane 250 down, in otherembodiments focal plane 250 may be shifted up and/or down as needed,using focus actuator 204 and/or lateral actuator 236 as needed.

FIG. 3 illustrates a flow chart of an exemplary method 300 forcomputational microscopic layer separation. In one example, each of thesteps shown in FIG. 3 may represent an algorithm whose structureincludes and/or is represented by multiple substeps, examples of whichwill be provided in greater detail below.

As illustrated in FIG. 3, at step 310 one or more of the systemsdescribed herein may measure, using the image sensor and theillumination system, one or more optical aberrations of the imagingdevice at the axial position. For example, controller 206 may measure,using image capture device 202 and illumination assembly 210, one ormore optical aberrations of microscope 200 at the axial position, whichmay correspond to focal plane 250.

The optical aberrations may manifest in various ways. In some examples,the optical aberrations may comprise a spherical aberration. In someexamples, the optical aberrations may comprise coma and/or astigmatism.In some examples, the optical aberrations may not comprise defocus. Inother examples, the optical aberrations may comprise two or more opticalaberrations, which may include defocus.

In some embodiments, one or more of image shifting, blurring or aspatially varying point spread function may be detected in acquiredimages in response to different illumination angles of the sample. Someaberrations, such as astigmatism and trefoil, can result in non-uniformimage shifting in response to the plurality of illumination angles,which can be used to detect the type and amount of aberration. Withastigmatism, the image shifting in response to the plurality ofillumination angles may occur greater in one direction than anotherdirection, e.g. asymmetrical, for example. With spherical aberration,such as fourth order and sixth order spherical aberration, the imageshifting may remain substantially symmetrical with respect to the changein illumination angle, for example. In some embodiments, the amount ofthe image shifting for defocus may correspond to the angle ofillumination. For example, a first illumination angle α1 closer to 90degrees will provide less image shift than a second illumination angleα2 that is farther from 90 degrees than 90 degrees.

In some embodiments, one or more of the types of aberrations or theamount of the one or more aberrations is determined in response to theone or more of image shifting, blurring or a spatially varying pointspread function. In some embodiments, the point spread function of theimaging system varies slowly across the image, and the amount ofaberrations is determined in response to the spatially varying pointspread function. In some embodiments, a portion of the image isanalyzed, such that the change in the image related to illuminationangle corresponds to one or more of the amount or type of the one ormore aberrations. In some embodiments, a plurality of portions isanalyzed to determine the localized aberrations for each of theplurality of portions of the image. Alternatively or in combination,each of the plurality of portions can be analyzed to determine change inimage structure in response to the plurality of illumination angles.Although reference is made to image shifting to determine one or more ofan amount or type of aberrations, in some embodiments the one or moreaberrations is determined without measuring image shifting, for examplewith an artificial intelligence algorithm such as one or more of machinelearning, a neural network, or a convolutional neural network and otherapproaches and combinations thereof as described herein, which candetect subtle change in image structure related to the one or moreaberrations in order to determine the one or more aberrations.

Optionally, at step 320 one or more of the systems described herein maycapture, using the image sensor, image data from a plurality of imagesat the axial position using the plurality of illumination angles. Forexample, controller 206 may capture, using image capture device 202,image data from multiple images at the axial position (e.g., focal plane250), using multiple illumination angles from illumination assembly 210.

In some examples, illumination assembly 210 may be configured toilluminate sample 214 (e.g., specimen 246) at a plurality ofillumination angles. For instance, illumination assembly 210 maycomprise a variable illumination source, such as an LED array positionedto illuminate sample 214 with one or more plane waves or approximateplane waves, from different angles at different times. Illuminationassembly 210 may illuminate sample 214 with more than one plane wave atthe same time, such as pairs of plane waves from different angles.

More specifically, illumination assembly 210 may include a plurality oflight sources. In some examples, controller 206 may illuminate thesample from the plurality of illumination angles simultaneously withsimultaneous activation of a plurality of light sources usingillumination assembly 210. In other examples, controller 206 mayilluminate the sample from the plurality of illumination anglessequentially with sequential activation of the plurality of light usingillumination assembly 210.

Optionally, at step 330 one or more of the systems described herein maycalculate an amount of shift in the image data between the plurality ofimages. For example, controller 206 may calculate the amount of shift inthe image data between the plurality of images. The shift in the imagedata and corresponding illumination angles can be used to determine theamount of the one or more aberrations as described herein.

In some examples, controller 206 may use one or more ofcross-correlation, a feature extraction and matching algorithm, and/or adata driven algorithm such as neural networks to detect an effectiveshift in each of the images acquired using illumination assembly 210 atdifferent angles of illumination.

In some examples, controller 206 may use knowledge of the properties ofthe optical system, such as numerical aperture (“NA”), and the effectiveshift calculated above to construct a set of linear and/or non-linearequations. Controller 206 may solve the set of equations to determinethe unknown aberrations.

In some examples, to reduce a number of unknowns in the set ofequations, controller 206 may know or partially know some of theaberrations and/or defocus. For example, sample 214 may be sufficientlydefocused in one direction such that a magnitude of the defocus isdetermined. In other examples, controller 206 may already know one ormore of the aberrations. For instance, controller 206 may comprisevalues corresponding to known amounts of aberrations except for defocusand spherical aberrations in order to achieve a more robustreconstruction via the equations. In another example, controller 206comprise values corresponding only a magnitude and/or sign of someaberrations. In yet other examples, controller 206 may have valuescorresponding to the sign of some aberrations such that the constructedequations may be non-linear.

In some embodiments, if it is a priori known, e.g. based on the systemconfiguration, that most of the aberrations are insignificant or known,and only some aberrations (e.g., defocus and/or spherical aberrations)are significant, controller 206 may consider only defocus and/orspherical aberrations as unknowns and consider the rest of theaberrations as known or ignored (if unknown) when solving the equations.The measured aberrations (which may or may not include defocus), may beused as the initial condition for computational microscopyreconstruction.

Optionally, at step 340 one or more of the systems described herein maydetermine, using the calculated amount of shift or other image data asdescribed herein, a distance between the sample and the focal plane. Forexample, controller 206 may determine, using the calculated amount ofshift, a distance between focal plane 250 and specimen 246.

In some examples, controller 206 may determine whether an object locatedat the focal plane is in focus in response to an image shift from imagescaptured with the image sensor and to adjust the axial position of thefocal plane in response to the object being in focus and an amount ofthe one or more aberrations above a threshold amount. As will beexplained further below, controller 206 may adjust focal plane 250 inresponse to dirt 248 being in focus and the amount of aberrationsexceeding a threshold amount. In some examples, controller 206 maydetermine the adjustment based in part on thickness 245 of coverslip244, which may be known, in order to adjust focal plane 250 intospecimen 246.

Although optional steps 320-340 are presented sequentially after step310 in FIG. 3, in some embodiments one or more of steps 320-340 maycomprise substeps of step 310. In other embodiments, one or more ofsteps 320-340 may be performed or skipped as appropriate.

Optionally, at step 350 one or more of the systems described herein maydetermine whether the focal plane is located on one or more of on asurface of a transparent material covering the sample, beneath a surfaceof the transparent material covering the sample or within the sample inresponse to an amount of the one or more aberrations. For example,controller 206 may determine whether focal plane 250 is located on asurface of coverslip 244, beneath the surface of coverslip 244, orwithin specimen 246, in response to the amount of aberrations.

In some examples, focal plane 250 may comprise an axial location withina depth of field of image capture device 202.

In some examples, image capture device 202 may be configured to imagesample 214 with the transparent material (e.g., coverslip 244) coveringsample 214 with a decreased amount of aberrations as compared to thetransparent material not covering sample 214. Controller 206 may beconfigured with instructions to adjust focal plane 250 toward sample 214(e.g., as seen the shift in focal plane 250 from FIG. 2A to FIG. 2B) inorder to decrease the one or more aberrations.

In some examples, image capture device 202 may be configured to image anobject with decreased aberrations with focal plane 250 of image capturedevice 202 located beneath an upper surface of the transparent coveringmaterial (e.g., coverslip 244) as compared to focal plane 250 located onor above the upper surface.

In some examples, controller 206 may be configured to determine whetherfocal plane 250 is located on dirt 248 on a surface of the transparentmaterial (e.g., coverslip 244) covering sample 214 (e.g., specimen 246)or on sample 214 beneath the surface of the covering material inresponse to the amount of the one or more aberrations and a structure ofan image captured with image capture device 202 at the axial position.For example, controller 206 may determine whether focal plane 250 islocated on dirt 248, as in FIG. 2A, or on specimen 246, as in FIG. 2B,in response to one or more of the amount of aberrations detected, or astructure of the image captured.

In some examples, sample 214 (e.g., specimen 246) may comprise a sparsesample. In some examples, the transparent material (e.g. coverslip 244)covering sample 214 may comprise a solid coverslip and/or a liquidcoverslip. In some examples, the transparent material covering sample214 may comprise an index of refraction of at least 1.4.

In some examples, controller 206 may be configured to determine theamount of the one or more aberrations in response to the calculatedamount of shift, as described herein. For example, the amount of the oneor more optical aberrations may be compared to a predetermined value ofthe one or more optical aberrations. Additionally, the axial positionmay be adjusted in response to the amount being greater than thepredetermined value or not adjusted in response to the amount being lessthan the predetermined value.

In some examples, controller 206 may be configured to determine whetherthe one or more optical aberrations correspond to a presence of atransparent material (e.g., coverslip 244) covering sample 214 betweenthe lens (e.g., objective lens 234 and/or tube lens 232) and focal plane250.

Although FIG. 3 illustrates optional step 350 as sequentially followingstep 340, in some embodiments step 350 may comprise a substep of any ofthe steps of method 300, such as a substep of step 310. In someembodiments, one or more of steps 320-340 may be performed or skippedbefore step 350. In some embodiments, step 350 may be performed beforeone or more of steps 320-340, in conjunction with one or more of steps320-340, or alternatively to one or more of steps 320-340.

At step 360 one or more of the systems described herein may determinewhether to adjust the focal plane with respect to the sample in responseto the one or more optical aberrations. For example, controller 206 maydetermine, based on the one or more optical aberrations, to adjust focalplane 250, such as the shift depicted between FIG. 2A and FIG. 2B asfurther described herein.

Although FIG. 3 illustrates a specific sequence of steps, in otherembodiments the steps may be performed in another order, simultaneously,skipped, or repeated, such as one or more of optional steps 320-350.

FIG. 4 illustrates a flow chart of an exemplary method 400 forcomputational microscopic layer separation. In one example, each of thesteps shown in FIG. 4 may represent an algorithm whose structureincludes and/or is represented by multiple substeps, examples of whichwill be provided in greater detail below.

As illustrated in FIG. 4, at step 410 one or more of the systemsdescribed herein may begin with a new FOV. For example, controller 206may adjust objective lens 234 and/or stage 216, using focus actuator 204and/or lateral actuator 236, to initial or default positions aftersample 214 is mounted onto stage 216 to start scanning. Although focalplane 250 should preferably be located at specimen 246, as in FIG. 2B,focal plane 250 may possibly be in an undesired position, as in FIG. 2A.In one example, controller 206 may have erroneously autofocused ontodirt 248, or otherwise incorrectly adjusted focal plane 250.

At step 420 one or more of the systems described herein may detect datain one layer (e.g., one focal plane). For example, controller 206, usingimage capture device 202, may capture data using a current position offocal plane 250. Depending on where focal plane 250 is positioned, thedata captured may be legitimate (e.g., in FIG. 2B), or may be unwanteddata (e.g., in FIG. 2A).

At step 430 one or more of the systems described herein may determine ifthe detected data is legitimate or dirt. For example, controller 206 maydetermine whether the detected data is legitimate (e.g., focal plane 250is located within specimen 246 as in FIG. 2B) or dirt (e.g., focal plane250 is located within dirt 248 as in FIG. 2A). Controller 206 maydetermine whether the detected data is legitimate by, for example asdescribed above, detecting a number of aberrations and comparing thenumber of detected aberrations to a threshold value of aberrations. Ifthe detected number exceeds the threshold value, the detected data maybe dirt, and method 400 may proceed to step 450. Otherwise, the detecteddata may be legitimate, and method 400 may proceed to step 440.

If the detected data is determined to be legitimate, then at step 440one or more of the systems described herein may stay in place, if thedetected data is legitimate data. For example, controller 206 may notshift focal plane 250 as focal plane 250 may already be located withinspecimen 246, as in FIG. 2B.

If the detected data is determined to be dirt, then at step 450 one ormore of the systems described herein may move down the coverslip, if thedetected data is dirt. For example, controller 206 may determine howmuch axial adjustment is needed to move focal plane 250 beyond coverslip244 and into specimen 246, as described above. Controller 206 mayaccordingly control focus actuator 204 to move objective lens 234 and/orlateral actuator 236 to move stage 216 such that focal plane 250 ismoved from dirt 248, as seen in FIG. 2A, into specimen 246, as seen inFIG. 2B.

At step 460 one or more of the systems described herein may continuescanning. For example, once focal plane 250 is at a desired positionwithin specimen 246, controller 206 may control image capture device 202to captured image data of specimen 246 and continue the scanning ofsample 214.

The steps of method 400 may be performed in any order and repeated asneeded. For example, multiple focal planes of specimen 246 may bescanned, and one or more of the desired focal planes may undergo method400.

As described herein, the computing devices and systems described and/orillustrated herein broadly represent any type or form of computingdevice or system capable of executing computer-readable instructions,such as those contained within the modules described herein. In theirmost basic configuration, these computing device(s) may each comprise atleast one memory device and at least one physical processor.

The term “memory” or “memory device,” as used herein, generallyrepresents any type or form of volatile or non-volatile storage deviceor medium capable of storing data and/or computer-readable instructions.In one example, a memory device may store, load, and/or maintain one ormore of the modules described herein. Examples of memory devicescomprise, without limitation, Random Access Memory (RAM), Read OnlyMemory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives(SSDs), optical disk drives, caches, variations or combinations of oneor more of the same, or any other suitable storage memory.

In addition, the term “controller”, “processor” or “physical processor,”as used herein, generally refers to any type or form ofhardware-implemented processing unit capable of interpreting and/orexecuting computer-readable instructions. In one example, a physicalprocessor may access and/or modify one or more modules stored in theabove-described memory device. Examples of physical processors comprise,without limitation, microprocessors, microcontrollers, CentralProcessing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) thatimplement softcore processors, Application-Specific Integrated Circuits(ASICs), portions of one or more of the same, variations or combinationsof one or more of the same, or any other suitable physical processor.The processor may comprise a distributed processor system, e.g. runningparallel processors, or a remote processor such as a server, andcombinations thereof.

Although illustrated as separate elements, the method steps describedand/or illustrated herein may represent portions of a singleapplication. In addition, in some embodiments one or more of these stepsmay represent or correspond to one or more software applications orprograms that, when executed by a computing device, may cause thecomputing device to perform one or more tasks, such as the method step.

In addition, one or more of the devices described herein may transformdata, physical devices, and/or representations of physical devices fromone form to another. Additionally or alternatively, one or more of themodules recited herein may transform a processor, volatile memory,non-volatile memory, and/or any other portion of a physical computingdevice from one form of computing device to another form of computingdevice by executing on the computing device, storing data on thecomputing device, and/or otherwise interacting with the computingdevice.

The term “computer-readable medium,” as used herein, generally refers toany form of device, carrier, or medium capable of storing or carryingcomputer-readable instructions. Examples of computer-readable mediacomprise, without limitation, transmission-type media, such as carrierwaves, and non-transitory-type media, such as magnetic-storage media(e.g., hard disk drives, tape drives, and floppy disks), optical-storagemedia (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), andBLU-RAY disks), electronic-storage media (e.g., solid-state drives andflash media), and other distribution systems.

A person of ordinary skill in the art will recognize that any process ormethod disclosed herein can be modified in many ways. The processparameters and sequence of the steps described and/or illustrated hereinare given by way of example only and can be varied as desired. Forexample, while the steps illustrated and/or described herein may beshown or discussed in a particular order, these steps do not necessarilyneed to be performed in the order illustrated or discussed.

The various exemplary methods described and/or illustrated herein mayalso omit one or more of the steps described or illustrated herein orcomprise additional steps in addition to those disclosed. Further, astep of any method as disclosed herein can be combined with any one ormore steps of any other method as disclosed herein.

The processor as described herein can be configured to perform one ormore steps of any method disclosed herein. Alternatively or incombination, the processor can be configured to combine one or moresteps of one or more methods as disclosed herein.

Unless otherwise noted, the terms “connected to” and “coupled to” (andtheir derivatives), as used in the specification and claims, are to beconstrued as permitting both direct and indirect (i.e., via otherelements or components) connection. In addition, the terms “a” or “an,”as used in the specification and claims, are to be construed as meaning“at least one of” Finally, for ease of use, the terms “including” and“having” (and their derivatives), as used in the specification andclaims, are interchangeable with and shall have the same meaning as theword “comprising.

The processor as disclosed herein can be configured with instructions toperform any one or more steps of any method as disclosed herein.

It will be understood that although the terms “first,” “second,”“third”, etc. may be used herein to describe various layers, elements,components, regions or sections without referring to any particularorder or sequence of events. These terms are merely used to distinguishone layer, element, component, region or section from another layer,element, component, region or section. A first layer, element,component, region or section as described herein could be referred to asa second layer, element, component, region or section without departingfrom the teachings of the present disclosure.

As used herein, the term “or” is used inclusively to refer items in thealternative and in combination.

As used herein, characters such as numerals refer to like elements.

The present disclosure includes the following numbered clauses.

Clause 1. A microscope comprising: at least one imaging devicecomprising lens and an image sensor; an illumination system forilluminating a sample; an actuator to adjust an axial position of afocal plane with respect to the sample; and a processor operativelycoupled to the at least one imaging device and the illumination system,the processor configured to: measure, using the image sensor and theillumination system, one or more optical aberrations of the imagingdevice at the axial position; and determine whether to adjust the focalplane with respect to the sample in response to the one or more opticalaberrations.

Clause 2. The microscope of clause 1, wherein the processor isconfigured to determine whether the focal plane is located on one ormore of on a surface of a transparent material covering the sample,beneath a surface of the transparent material covering the sample orwithin the sample in response to an amount of the one or moreaberrations.

Clause 3. The microscope of clause 2, wherein the imaging device isconfigured to image the sample with the transparent material coveringthe sample with a decreased amount of aberrations as compared to thetransparent material not covering the sample and wherein the processoris configured with instructions to adjust the focal plane toward thesample in order to decrease the one or more aberrations.

Clause 4. The microscope of clause 2, wherein the imaging device isconfigured to image an object with decreased aberrations with the focalplane of the imaging device located beneath an upper surface of thetransparent material as compared to the focal plane located on or abovethe upper surface.

Clause 5. The microscope of clause 2, wherein the processor isconfigured to determine whether the focal plane is located on dirt on asurface of the transparent material covering the sample or on the samplebeneath the surface of the transparent material in response to theamount of the one or more aberrations and a structure of an imagecaptured with the at least one imaging device at the axial position.

Clause 6. The microscope of clause 5, wherein the sample comprises asparse sample.

Clause 7. The microscope of clause 2, wherein the transparent materialcovering the sample comprises one or more of a solid coverslip or aliquid coverslip.

Clause 8. The microscope of clause 2, wherein the transparent materialcovering the sample comprises an index of refraction of at least 1.4.

Clause 9. The microscope of clause 1, further comprising an illuminationsystem to illuminate a sample at a plurality of illumination angles.

Clause 10. The microscope of clause 9, wherein the processor isconfigured to: capture, using the image sensor, image data from aplurality of images at the axial position using the plurality ofillumination angles; calculate an amount of shift in the image databetween the plurality of images; and determine, using the calculatedamount of shift, a distance between the sample and the focal plane.

Clause 11. The microscope of clause 10, wherein the processor isconfigured to determine the amount of the one or more aberrations inresponse to the calculated amount of shift.

Clause 12. The microscope of clause 10, wherein the processor isconfigured to illuminate the sample from the plurality of illuminationangles simultaneously with simultaneous activation of a plurality oflight sources.

Clause 13. The microscope of clause 12, wherein the processor isconfigured to illuminate the sample from the plurality of illuminationangles sequentially with sequential activation of the plurality of lightsources.

Clause 14. The microscope of clause 1, wherein the amount of the one ormore optical aberrations is compared to a predetermined value of the oneor more optical aberrations and optionally wherein the axial position isadjusted in response to the amount being greater than the predeterminedvalue or not adjusted in response to the amount being less than thepredetermined value.

Clause 15. The microscope of clause 1, wherein the processor isconfigured to determine whether the one or more optical aberrationscorrespond to a presence of a transparent material covering the samplebetween the lens and the focal plane.

Clause 16. The microscope of clause 1, wherein the processor isconfigured with instructions to determine whether an object located atthe focal plane is in focus in response to an image shift from imagescaptured with the image sensor and to adjust the axial position of thefocal plane in response to the object being in focus and an amount ofthe one or more aberrations above a threshold amount.

Clause 17. The microscope of clause 1, wherein the one or more opticalaberrations comprises a spherical aberration.

Clause 18. The microscope of clause 17, wherein the one or more opticalaberrations comprises one or more of coma or astigmatism.

Clause 19. The microscope of clause 1, wherein the one or more opticalaberrations does not comprise defocus.

Clause 20. The microscope of clause 1, wherein the one or more opticalaberrations comprise two or more optical aberrations and wherein the twoor more optical aberrations include defocus.

Clause 21. The microscope of clause 1, wherein the focal plane comprisesan axial location within a depth of field of the at least one imagingdevice.

Clause 22. A method comprising: illuminating a sample with anillumination system; imaging the sample with at least one imaging devicecomprising lens and an image sensor; adjusting, with an actuator, anaxial position of a focal plane with respect to the sample; andmeasuring, using the image sensor and the illumination system, one ormore optical aberrations of the imaging device at the axial position;and determining whether to adjust the focal plane with respect to thesample in response to the one or more optical aberrations.

Embodiments of the present disclosure have been shown and described asset forth herein and are provided by way of example only. One ofordinary skill in the art will recognize numerous adaptations, changes,variations and substitutions without departing from the scope of thepresent disclosure. Several alternatives and combinations of theembodiments disclosed herein may be utilized without departing from thescope of the present disclosure and the inventions disclosed herein.Therefore, the scope of the presently disclosed inventions shall bedefined solely by the scope of the appended claims and the equivalentsthereof

What is claimed is:
 1. A microscope comprising: at least one imaging device comprising lens and an image sensor; an illumination system for illuminating a sample; an actuator to adjust an axial position of a focal plane with respect to the sample; and a processor operatively coupled to the at least one imaging device and the illumination system, the processor configured to: measure, using the image sensor and the illumination system, one or more optical aberrations of the imaging device at the axial position; and determine whether to adjust the focal plane with respect to the sample in response to the one or more optical aberrations.
 2. The microscope of claim 1, wherein the processor is configured to determine whether the focal plane is located on one or more of on a surface of a transparent material covering the sample, beneath a surface of the transparent material covering the sample or within the sample in response to an amount of the one or more aberrations.
 3. The microscope of claim 2, wherein the imaging device is configured to image the sample with the transparent material covering the sample with a decreased amount of aberrations as compared to the transparent material not covering the sample and wherein the processor is configured with instructions to adjust the focal plane toward the sample in order to decrease the one or more aberrations.
 4. The microscope of claim 2, wherein the imaging device is configured to image an object with decreased aberrations with the focal plane of the imaging device located beneath an upper surface of the transparent material as compared to the focal plane located on or above the upper surface.
 5. The microscope of claim 2, wherein the processor is configured to determine whether the focal plane is located on dirt on a surface of the transparent material covering the sample or on the sample beneath the surface of the transparent material in response to the amount of the one or more aberrations and a structure of an image captured with the at least one imaging device at the axial position.
 6. The microscope of claim 5, wherein the sample comprises a sparse sample.
 7. The microscope of claim 2, wherein the transparent material covering the sample comprises one or more of a solid coverslip or a liquid coverslip.
 8. The microscope of claim 2, wherein the transparent material covering the sample comprises an index of refraction of at least 1.4.
 9. The microscope of claim 1, further comprising an illumination system to illuminate a sample at a plurality of illumination angles.
 10. The microscope of claim 9, wherein the processor is configured to: capture, using the image sensor, image data from a plurality of images at the axial position using the plurality of illumination angles; calculate an amount of shift in the image data between the plurality of images; and determine, using the calculated amount of shift, a distance between the sample and the focal plane.
 11. The microscope of claim 10, wherein the processor is configured to determine the amount of the one or more aberrations in response to the calculated amount of shift.
 12. The microscope of claim 10, wherein the processor is configured to illuminate the sample from the plurality of illumination angles simultaneously with simultaneous activation of a plurality of light sources.
 13. The microscope of claim 12, wherein the processor is configured to illuminate the sample from the plurality of illumination angles sequentially with sequential activation of the plurality of light sources.
 14. The microscope of claim 1, wherein the amount of the one or more optical aberrations is compared to a predetermined value of the one or more optical aberrations and optionally wherein the axial position is adjusted in response to the amount being greater than the predetermined value or not adjusted in response to the amount being less than the predetermined value.
 15. The microscope of claim 1, wherein the processor is configured to determine whether the one or more optical aberrations correspond to a presence of a transparent material covering the sample between the lens and the focal plane.
 16. The microscope of claim 1, wherein the processor is configured with instructions to determine whether an object located at the focal plane is in focus in response to an image shift from images captured with the image sensor and to adjust the axial position of the focal plane in response to the object being in focus and an amount of the one or more aberrations above a threshold amount.
 17. The microscope of claim 1, wherein the one or more optical aberrations comprises a spherical aberration.
 18. The microscope of claim 17, wherein the one or more optical aberrations comprises one or more of coma or astigmatism.
 19. The microscope of claim 1, wherein the one or more optical aberrations does not comprise defocus.
 20. The microscope of claim 1, wherein the one or more optical aberrations comprise two or more optical aberrations and wherein the two or more optical aberrations include defocus.
 21. The microscope of claim 1, wherein the focal plane comprises an axial location within a depth of field of the at least one imaging device.
 22. A method comprising: illuminating a sample with an illumination system; imaging the sample with at least one imaging device comprising lens and an image sensor; adjusting, with an actuator, an axial position of a focal plane with respect to the sample; and measuring, using the image sensor and the illumination system, one or more optical aberrations of the imaging device at the axial position; and determining whether to adjust the focal plane with respect to the sample in response to the one or more optical aberrations. 